NASA Technical Reports Server (NTRS) 20150000376: Development, Validation, and Application of OSSEs at NASA/GMAO

Development, Validation, and Application of
OSSEs at NASA/GMAO

Ronald Errico
Nikki Prive

Goddard Earth Sciences Technology and Research Center

at Morgan State University

and

Global Modeling and Assimilation Office
at NASA Goddard Space Flight Center

GESTAR

MORGAN STATE UNIVERSITY-

Acknowledgements

Runhua Yang
Ricardo Todling
Meta Sienkiewicz
Jing Guo
Will McCarty
Joseph Stassi
Wei Guo
Arlindo Da Silva
Amal El Akkraoui
Joanna Joiner
Ronald Gelaro
Michele Rienecker
ECMWF

Outline:

1 . General methodology and requirements

2. Simulation of observations and their errors

3. OSSE validation in terms of DAS statistics

4. OSSE validation in terms of forecast statistics

5 . Warnings

6. Problems with latest GMAO experiments

Data Assimilation of Real Data

Time

Observing System Simulation Experiment

Applications of OSSEs

1. Estimate effects of proposed instruments (and their competing
designs)on analysis skill by exploiting simulated environment.

2. Evaluate present and proposed techniques for data assimilation
by exploiting known truth.

Requirements for an OSSE

1 . A simulation of “truth”, generally produced as a free-running
forecast produced by an NWP model (termed the “Nature Run”).

2. A simulation of a complete set of observations, drawn from truth.

3. A simulation of observational instrument plus representativeness
errors to be added to the results of step 2.

4. A data assimilation system to ingest the results of step 3.

All these steps must be done well enough to produce believable and
useful metrics from step 4.

Choice of a Nature Run

1 . A good simulation of nature in all important aspects

2. Ideally, individual realizations of the NR should be
indistinguishable from corresponding realizations of
nature (e.g., analyses) at the same time of year.

3. Since a state-of-the-art OSSE will require a cycling DAS,
the NR should have temporal consistency.

4. For either 4DVAR or FGAT 3D VAR, or for high spatial resolution,
NR datasets should have high frequency (i.e. ,<6 hours)

5. Since dynamic balance is an important aspect of the
atmosphere affecting a DAS, the NR datasets should
have realistic balances.

6. For these and other reasons, using a state-of-the-art NWP
model having a demonstrated good climatology to produce
NR data sets is arguably the best choice.

Simulations of Nature (Nature Runs)

NR Source

ECMWF

GMAO

Availability

now

Summer 2014

Horizontal Resolution

35 km

7 km

# Vertical levels

91

72

Full Period

1 year

2 year

Output Frequency

3 hourly

0.5 hourly

Other Characteristics

03

16 aerosols, CO, C02, 03

Data set size

2 TB

2500 TB

Simulation of Observations

1 . Any observation that can be assimilated can be simulated!

| y = H z { z)|

2. Differences between the H used to assimilate and simulate
will be interpreted by the DAS as representativeness errors

t R = H(x f [z]) -H z { z)

3. Therefore, as more realism is modeled for the observation
simulation compared to the assimilation, more
representativeness error is introduced, including gross errors
that must be identified by the DAS quality control.

4. It may be necessary to compensate for deficiencies in the nature
run (e.g., unrealistic cloud cover) when simulating observations.

Simulation of Observation Errors

1 . When simulating observations, there is no instrument and
thus no instrument error. It therefore needs to be explicitly
simulated and added, unless it is negligible.

2. An error of representativeness is generally implicitly created:

e R = H(x t [ z]) - H z ( z)

3. The real representativeness error is likely underestimated by

the implicit component. Additional representativeness error needs
to be simulated.

4. If real observation errors are correlated, the simulated ones
must also be if the OSSE is to verify.

5. Errors effectively removed by the DAS may not require careful
simulation!

Probabilities of radiances being affected by clouds

Effective radiative surface is a high-level cloud

Ph= f P(H\f h )P(f h ) df h

JF h

Effective radiative surface is a medium-level cloud

Pm = [ f P([M\H]\fm ) [1 - P(H\f h )] P(fmJh) dfm dfh

J F h J F m

Effective radiative surface is a low-level cloud

Pi, = i [ f P([L\H n M]\fi) [1 - P(H\f h ) - P(M\f m , //,)] P(fi J m , fh) df, df m df h

JF h JF m JF,

P(H | f h )

Channel

AIRS

Locations of QC-accepted observations for AIRS channel 295 at 18 UTC 12 July

Simulated

Real

Simulation of observations

RAOB observation lunch time and final pressure determined by
corresponding real observation

RAOB trajectory determined from NR wind fields

RAOB significant levels determined from NR sounding

SATWINDS determined from locations with NR clouds or
moisture gradients

All relevant metrics depend on system errors

Analysis error

A = Ai B.B.R.R. Q)

Background or Forecast error error

B = B( A, Q, M)

Observation error

R = R(E, F)

Daley and Menard 1 993 MWR

Observational Error Simulation

All observations have added random error with tuned variances
Portions of added errors for:

RAOB soundings are vertically correlated
AMSUA, MHS are horizontally correlated
SAT WINDS vertically and horizontally correlated
AIRS and IASI channel correlated
No mean errors added

No gross errors added

Assimilation System

GEOS-5 (GMAO) DAS/Model
NCEP/GMAO GSI (3DVAR) scheme
Resolution: 55km, 72 levels

Evaluation for 1-31 July 2005, with 2 week, accelerated spin up in June.

Observations include all “conventional” observations available in 2011
(except for precipitation rates) and all radiance available from AMSU-A,
MHS, HIRS-4, AIRS, IASI instruments, plus GPSRO.

Validation of OSSEs

As for any simulation, OSSE results apply to the assimilation
of real data only to the degree the OSSE for such an application
validates well with regard to relevant metrics.

OSSE validity is first determined by carefully comparing a
variety of statistics that can be computed in both the real
assimilation and OSSE contexts.

Since data assimilation is a fundamentally statistical problem,
OSSE validation and application must generally be statistical.

Std. Dev. 0-F

Standard deviations of QC-accepted y-H(xb) values (Real vs. OSSE)

RAOB U Wind

AMSU-AMETOP-A

Channel

Correlation

Horizontal correlations of y-H(xb)
Evaluations for 20-90 N

GOES-IR SATWND 300 hPa RAOB T 700 hPa

is OSSE without correlated observation errors

90

80

70

60

50

40

30

20

10

Correlations Between Channels of AIRS Innovations

OSSE

REAL

T 850 hPa

Time mean
Analysis increments

U 500 hPa

- 180-120 -60

0 60 120 180
Longitude

- 180-120 -60

0 60 120 180
Longitude

OSSE

30

0

-30

Real

■ - 0.5
.

-1

- 180-120 -60 0 60 120 180

Longitude

-3

- 180-120 -60 0 60 120 180
Longitude

p, hPa p, hPa

Square roots of zonal means of temporal variances of analysis increments

100
200
300
400
500
600
700
800
900

-90 -60 -30 0 30 60 90
Latitude

T Real

0.6
0.5
0.4
0.3
0.2
0.1

-90 -60 -30 0 30 60 90

Latitude

Anomaly Correlation

OSSE vs Real Data: Forecasts

NH Anomaly Correlation: July SH Anomaly Correlation: July

U-Wind RMS error: July

U, 30N-90N U, 30S-90S U, 30S-30N

Solid lines: 24 hour RMS error vs analysis
Dashed lines: 120 hr forecast RMS error vs analysis

July Adjoint: dry error energy norm

Adjoint Observation Impact

Radar W
AMSU-B
AMSU-A
MSU
AIRS
HIRS
RAOBW
RAOBQ
RAOBT
ocean wind
aircraft W
aircraft T
Satwind
surf W
surf Q
surf T
ocean Ps
land Ps

0

Control

OSSE

- 0.1

- 0.2

- 0.3 - 0.4

Observation Impact

- 0.5

- 0.6

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

0 .

Adjoint:

dry error energy norm

Adjoint AMSU-A Impact

1

1 Control
^■OSSE

- 0.02 - 0.04 - 0.06 - 0.08 - 0.1 - 0.12 - 0.14 - 0.16 - 0.18

Observation Impact

Warnings

Past problems with some OSSEs

1 . Some OSSEs use a very reduced observation set of as a control

2. Some OSSEs have very limited validation

3. Some OSSEs are based on very limited “case studies”

4. Some OSSEs use unrealistic observation errors (e.g., no rep. error)

5. Some OSSEs use a deficient NR

Warnings

General criticisms of OSSEs

1 . In OSSEs, the NR and DAS models are generally too alike,
therefore underestimating model error and yielding
overly-optimistic results.

2. When future specific components of the observing systems
are deployed, the system in general will be different as will
the DAS techniques, and therefore the specific OSSE results
will not apply.

3. OSSEs are just bad science!

Response to Warnings

1. Design OSSEs thoughtfully.

2. Consider implications of all assumptions.

3. Validate OSSEs carefully.

4. Beware of quick shortcuts.

5. Specify reasonable observation error statistics.

6. Avoid conflicts of interest.

7. Avoid over-selling results.

8. Only attempt to answer appropriate questions.

9. Be skeptical of others’ works.

10. Be critical of your own work (This is science!).

Fractional reduction of zonal means of temporal variances of analysis errors

compared with background errors

T

9 9

— p*

,2

u

100

200

300

ns 400

CL
sz

® 500

w

2 600
a.

ns

LLJ 700

800

900

1000

*

I

I

A

-90 -60 -30 0 30 60

Latitude

90

-50 0 50

Latitude

0.25

0.2

0.15

0.1

0.05

0

- 0.05

-0.1

- 0.15

- 0.2

- 0.25

From validation experiment reported last year

Fractional reduction of zonal means of temporal variances of analysis errors

compared with background errors

ou DU -90 -60 -30 0

Latitude

From latest validation experiment

Square roots of zonal means of temporal variances of analysis increments

T Real

U Real

700

800

900

000 '

-90

T OSSE

-90 -60 -30 0 30 60 90

U OSSE

-90 -60 -30 0 30 60 90

Characteristics of Real Observation Errors

1 . Generally unknown

2. Even statistics not well known

3. Often biased

4. Correlated (maybe even with background error)

5. Include gross errors

6. Generally non-Gaussian

(a result of 5 or some basic physics; e.g. nonlinearity)

GMAO OSSE Validation Publications

Errico, R. M., R. Yang, N. C. Prive, K.-S. Tai, R. Todling, M. E. Sienkiewicz,

J. Guo, 2013: Development and validation of observing system simulation
experiments at NASA’s Global Modeling and Assimilation Office.

Quart. J. Roy. Meter. Soc., 139 , 1162-1178.

Prive, N. C., R. M. Errico, K.-S. Tai, 2013: Validation of forecast skill of the
Global Modeling and Assimilation Office observing system simulation experiment.
Quart. J. Roy. Meteorol. Soc., 139 , 1354-1363.